System and method for compensation of relection on a display device

ABSTRACT

A system and method for compensating for reflections caused by light-generating objects in the scene facing a display device includes capturing images of the scene. Reflection-inducting zones corresponding to the light generating objects are identified from the captured images. The reflection effect on the display device from the reflection-inducing zones are estimated. A target image to be displayed on the display device is adjusted based on the estimated reflection effect.

RELATED PATENT APPLICATION

The present application claims priority from U.S. provisional patentapplication No. 62/436,667, filed on Dec. 20, 2016, and entitled SYSTEMAND METHOD FOR COMPENSATION OF REFLECTION ON A DISPLAY DEVICE, thedisclosure of which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The technical field generally relates to performing digital imageprocessing to compensate for the reflection of one or more specularreflections within an ambient environment.

BACKGROUND

When light interacts with the display device of a mobile device, thelight may be reflected in multiple ways. For example, reflectance can bequantified in terms of diffuseness of reflection, varying between fullydiffuse to fully specular. For example, shining a spot light on aperfect mirror will provide only specular reflection such that the spotlight will only be visible in the reflected image when the mirror isdisplayed from an angle complimentary to the angle of the spot light.That is, if the spot light is located to the left of center of themirror and shown onto the mirror from 45 degrees, the spot light willonly be visible when it is viewed by an observer located 45 degrees tothe right of the mirror. On the other hand, diffuse surfaces will acceptlight and reflect it in all angles. Therefore, if a spot light is shownon a fully diffuse surface, some of the light from the spotlight can beviewed from all angles with the reflector appearing bright that it doeswhen the spotlight is not illuminating the diffuser. Most displays arenot perfect mirror or perfect diffusers, but instead reflect some lightspecularly, like the mirror, and some light diffusely. The totalreflection, including both the diffuse and specular reflection providedby the display can be characterized using the bidirectional reflectiondistribution function.

Modern display devices, especially those found on mobile devices, tendto show specular reflections in high-ambient lighting situations. Asseen in FIGS. 1A and 1B, such reflections make it difficult to viewcontent, and the shifting highlights corresponding to background changesand screen tilting further confounds a person's view, particularly forvideo.

A commonly applied method to mitigate the effect of ambient involvesraising the black level, but this only serves to further underminecontrast and does little to compensate for the confound betweendisplayed content and reflected background information.

Compensation for low-light conditions can be achieved by manipulatingcolor and tone-mapping appropriately. However, bright ambientenvironments pose a number of difficulties, including screenreflections, surrounding glare, and limited display brightness. To someextent, the first two problems can be solved by eliminating the thirdone, i.e., creating a brighter display to overcome the ambientenvironment. The unacceptable price in a mobile device is powerconsumption, but for automotive, it is mostly a question oftechnological limits. Displays only get so bright before special coolingis required with the latest LED-backlit LCD or OLED panels. Furthermore,such adjustment better addresses diffuse reflection than specularreflection Nevertheless, increasing brightness will probably be thefirst resort to address readability issues in automotive displays. See,for example, U.S. Pat. Nos. 6,411,306, 7,545,397.

Reducing screen reflection is another well-known approach, typicallyusing low-reflectance coatings. While strides have been made inthin-film coatings for touch screens, and meta-materials hold promisefor further improvements, screen reflections will never be eliminatedcompletely when illumination sources are orders of magnitude brighterthan the display. A specular reflectance as low as 0.01%, which is anorder of magnitude better than any existing coatings, will result inindirect solar reflections (bounced off a car, a pane of glass, or bodyof water) above 10,000 nits. The problem is simply too large to go awayunder all conditions.

SUMMARY OF THE INVENTION

According to one aspect, there is provided a method for compensating forreflection on a display device. The method includes capturing one ormore images of a scene facing the display device, identifying from thecaptured images one or more reflection-inducing zones located within thescene facing the display device, determining specular reflection effecton the display device caused by the reflection-inducing zones, andadjusting a target image to be displayed on the display device based onthe determined reflection effect.

According to various aspects, a computer-implemented system includes atleast one data storage device; and at least one processor operablycoupled to the at least one storage device, the at least one processorbeing configured for performing the methods described herein accordingto various aspects.

According to one example embodiment, a computer-implemented systemincludes at least one data storage device; and

at least one processor operably coupled to the at least one storagedevice, the at least one processor being configured for

receiving one or more captured images of a scene facing the displaydevice;

identifying from the captured images one or more reflection-inducingzones located within the scene facing the display device;

determining specular reflection effect on the display device caused bythe reflection-inducing zones; and

adjusting a target image to be displayed on the display device based onthe determined specular reflection effect.

According to various aspects, a computer-readable storage mediumincludes computer executable instructions for performing the methodsdescribed herein according to various aspects.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the embodiments described herein and toshow more clearly how they may be carried into effect, reference willnow be made, by way of example only, to the accompanying drawings whichshow at least one exemplary embodiment, and in which:

FIG. 1a illustrates a first sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice;

FIG. 1b illustrates a second sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice;

FIG. 2 illustrates a flowchart of the operational steps of an examplemethod for compensating for reflection on a display device;

FIG. 3 illustrates a flowchart of the operational steps of an examplemethod for identifying one or more reflection-inducing zones;

FIG. 4 illustrates a flowchart of the operational steps of an examplemethod for determining distance of light generating objects in the scenefrom the display device;

FIG. 5 illustrates a flowchart of the operational steps of an examplemethod for determining the reflection effect on the display device;

FIG. 6 illustrates a schematic diagram showing the relative positions ofa viewer, the display device, and an image capture device;

FIG. 7 illustrates a flowchart of the operational steps of an examplemethod for adjusting the target image to be displayed based on thereflection effect;

FIG. 8a illustrates the first sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice in which compensation for reflection is mismatched;

FIG. 8b illustrates the second sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice in which compensation for reflection is mismatched;

FIG. 8c illustrates the first sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice in which reflection compensation according to various exemplaryembodiments described herein has been applied;

FIG. 8d illustrates the second sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice in which reflection compensation according to various exemplaryembodiments described herein has been applied;

FIG. 8e illustrates the first sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice in which reflection compensation according to various exemplaryembodiments described herein has been applied;

FIG. 8f illustrates the second sample image showing reflection effect onthe display device caused by a window in the scene facing the displaydevice in which reflection compensation according to various exemplaryembodiments described herein has been applied;

FIG. 9a is an image captured of the scene facing the display device;

FIG. 9b is a mask showing reflection zones (in white) determined fromthe image of the scene facing the display device;

FIG. 10a is a third sample image prior to processing;

FIG. 10b illustrates the third sample image after adjustment forreflection compensation;

FIG. 10c illustrates the displayed third sample image showing reflectioneffect and without reflection compensation; and

FIG. 10d illustrates the displayed third sample image in whichreflection compensation according to various exemplary embodimentsdescribed herein has been applied.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity.

DETAILED DESCRIPTION

Broadly described, various example embodiments described herein providefor a system and method for compensating for reflections caused bylight-generating objects in the scene facing a display device bycapturing images of the scene, identifying in the imagesreflection-inducing zones corresponding to the light generating objects,estimating the reflection effect on the display device from thereflection-inducing zones and adjusting a target image to be displayedbased on the estimated reflection effect. The reflection-inducing zonesmay be zones that cause specular reflection and the estimating estimatesthe specular reflection effect on the display device.

One or more reflection compensation systems and methods described hereinmay be implemented in computer programs executing on programmablecomputers, each comprising at least one processor, a data storage system(including volatile and non-volatile memory and/or storage elements), atleast one input device, and at least one output device. For example, andwithout limitation, the programmable computer may be a programmablelogic unit, a mainframe computer, server, and personal computer, cloudbased program or system, laptop, personal data assistance, cellulartelephone, smartphone, wearable device, tablet device, virtual realitydevices, smart display devices (ex: Smart TVs), set-top box, video gameconsole, or portable video game devices.

Each program is preferably implemented in a high level procedural orobject oriented programming and/or scripting language to communicatewith a computer system. However, the programs can be implemented inassembly or machine language, if desired. In any case, the language maybe a compiled or interpreted language. Each such computer program ispreferably stored on a storage media or a device readable by a generalor special purpose programmable computer for configuring and operatingthe computer when the storage media or device is read by the computer toperform the procedures described herein. In some embodiments, thesystems may be embedded within an operating system running on theprogrammable computer. In other example embodiments, the system may beimplemented in hardware, such as within a CPU or video card (GPU).

Furthermore, the systems, processes and methods of the describedembodiments are capable of being distributed in a computer programproduct comprising a computer readable medium that bears computer-usableinstructions for one or more processors. The medium may be provided invarious forms including one or more diskettes, compact disks, tapes,chips, wireline transmissions, satellite transmissions, internettransmission or downloadings, magnetic and electronic storage media,digital and analog signals, and the like. The computer-usableinstructions may also be in various forms including compiled andnon-compiled code.

The one or more reflection compensation system and methods describedherein is applied where an image or video (hereinafter referred to as a“target image”) is to be displayed on an electronic display device.Without limitation, the electronic display device may be a computermonitor, a screen of mobile device (ex: tablet, smartphone, laptop,wearable device), a screen of video game console, a TV, etc. The displaydevice may be implemented using display technologies such as OLED, LCD,quantum dot display, laser projector, CRT, etc.

FIGS. 1a and 1b show a first and second sample image representing acommon reflection problem. Light-generating objects in the area facingthe display device causes reflection on the display device that appearas whitish highlights on the display device.

Referring now to FIG. 2, therein illustrated is a flowchart of theoperational steps of an example method 100 for compensating for specularreflection on a display device.

At step 108, one or more images of a scene facing the display device iscaptured. The display device is to be used to display one or more targetimages. The scene facing the display device corresponds to theenvironment in front of the display device.

Any object in the scene that emits light that causes specular reflectionon the display device is referred herein generally as a“light-generating object”. It will be understood that thelight-generating object may directly emit light that causes specularreflections, or the light-generating object may be reflecting light froman external source, the reflected light further causing specularreflections on the display device.

The one or more images may be captured sequentially, such as in a video.Accordingly, the scene being captured in the images may change over thesequence, such as due to a change in orientation of the image capturedevice or changes to objects in the scene.

The one or more images of the scene can be captured by an image capturedevice. The image capture device may be positioned to be offset by aknown distance and orientation from the display device. Preferably, theimage capture device is located in proximity of the display device andis facing the same direction as the display device. For example, theimage capture device may be an external camera positioned in proximityof the display device. Alternatively, the image capture device may be anembedded camera, such as the front facing camera of a mobile device(smartphone, tablet, laptop with webcam, video game console, etc.).

The image capture device may be a combination of capture devices, suchas a combination of a camera and an ambient light sensor. Preferably,the camera and the ambient light sensor are located in proximity of oneanother such that a scene captured by the camera substantiallycorresponds to the scene captured by the ambient light sensor. It willbe appreciated that various mobile devices are now offered with both acamera and an ambient light sensor.

In some example embodiments, the image capture devices may have two ormore cameras, which may be operated to capture the scene while providingdepth information of objects within the scene (ex: stereoscopiccameras). An additional device operable to determine depth, such as atime-of-flight sensor, can also be used.

The images captured of the scene may be down-sampled to a lowerresolution, which may improve processing speed. It will be appreciatedthat steps described herein that operate on captured images of the scenefacing the display can refer to the down-sampled captured images.

The scene facing the display device represented in the images capturedby the image capture device is defined by the field of view of thecapture device.

Continuing with FIG. 2, at step 116, one or more reflection-inducingzones located within the one or more captured images are identified. Thereflection-inducing zones correspond to light-generating objects in thescene that can cause specular reflections on the display device. Withinthe captured images, the reflection-inducing zones are areas of thecapture images that have a sufficiently high luminance value thatindicates bright light-generating objects in the scene.

At step 124, the specular reflection effect on the display device causedby the light-generating objects, and as represented by thereflection-inducing zones in the captured images, are determined. Thespecular reflection effect represents an estimation of how a viewerviewing the display device would perceive specular reflections caused bylight-generating objects in the scene facing the display device.

At step 132, a target image that is to be displayed on the displaydevice is adjusted based on the reflection effect. The target image maybe adjusted to reduce or mitigate the reflections perceived by theviewer. The adjustment may include digitally processing the targetimage.

Referring now to FIG. 3, therein illustrated is a flowchart of theoperational steps of an example method for identifying one or morereflection-inducing zones located within the one or more capturedimages. The method may correspond to substeps of step 116.

At step 208, an area of the one or more captured images that is not areflection-inducing zone is identified. This zone corresponds to a partof the scene facing the display device that will not cause significantspecular reflections to be perceived by the viewer.

In one example, the face of the viewer is captured within the images ofthe scene facing the display device and zone of the images correspondingto a portion of the face of the viewer is used as a reference area toset a threshold for identifying reflection-inducing zones of thecaptured images. For example, an area of the face corresponding to thebridge of the viewer's nose may be used. The area may also include partsof the forehead and portions of each eye of the user.

At step 216, a threshold for determining reflection-inducing zones isset. The threshold may be set as a luminance value that is a multiple(ex: 100 times) of the average of the luminance values of the pixelsforming the reference area within the captured images. Alternatively, oradditionally, the threshold may be set as a factor of a maximum imagevalue (ex: high luminance in the reference area such that the thresholdexceeds maximum luminance pixel values).

At step 224, the areas of the captured image that have luminance valuesthat exceed the threshold are determined to be reflection-inducingzones. In one example, a smoothing or dilating may be applied to removereflection-inducing zones below a certain size.

At step 232, the distances of each light-generating object representedby the reflection-inducing zones from the display device are determined.That is, for each reflection-inducing zone identified from step 224, thedistance of the real-life light-generating object represented by thatreflection-inducing zone is determined.

According to one example, the distances of each light-generating objectmay be determined from known properties of the scene facing the displaydevice. This may be the case where the scene is fixed relative to thedisplay device. The location of each light-generating object in thescene, including their distance from the display device, can bepredefined and the identified reflection-inducing zones are matched toits corresponding light-generating object.

For example, a display device, such as computer monitor or a TV may bein a fixed position in a space, such as within a room, andlight-generating objects found within the space (ex: walls, windows,light fixtures, lamps) are predefined. The display device in a fixedposition may be an electronic billboard or other display devicepositioned in a public space.

For example, a display device may be in a fixed position inside theinterior cabin of an automobile and light-generating objects of thecabin (ex: windows of the vehicle, lights inside the cabin) arepredefined. It will be appreciated that although the automobile ismovable, the display device remains in a fixed position relative to theinterior of the cabin.

In one or more examples, properties of the scene, such as location andbrightness of light-generating objects, may be known ahead of time. Forexample, such properties may be pre-measured and pre-stored.

According to one example embodiment, the distance of light-generatingobjects within the scene from the display device is determined fromsensed motion of the display device and the movement ofreflection-inducing zones within images captured of the scene as thedisplay device is moved. The distance of the light-generating objectscan be calculated based on parallax effect.

The determining of distances of light generating objects within thescene may take into account the offset of the image capture device fromthe display device. Additionally or alternatively, the determining ofthe distances of the light generating objects within the scene may takeinto account the position of the viewer, such as the viewer's eyes.

Referring now to FIG. 4, therein illustrated is a flowchart showing theoperational steps of an example method 232 for determining distance oflight-generating objects in the scene from the display device.

At step 308, the motion of the image capture device is sensed. Themotion may be sensed by a sensor external to the image capture devicebut that is representative of the motion of the image capture device.For example, where the display device is a mobile device (ex:smartphone, tablet, laptop, portable video console), the motion may besensed with a motion sensor of the mobile device, such as a gyroscope.As the image capture device is being moved, a plurality of images of thescene facing the image capture device is captured.

At step 316, the movement of reflection inducing zones within the sceneis determined from the plurality of images of the scene captured duringmovement of the image capture device.

At step 324, the sensed motion of the image capture device is correlatedwithin the determined movement of reflection inducing zones to determinethe distance of light-generating objects represented by the reflectioninducing zones from the display device.

According to one example embodiment, lateral movement of the imagecapture device is sensed. Furthermore, edges of the reflection-inducingzones that are approximately perpendicular to the direction of motionare identified. The movement of the edges within the scene representedby the plurality of captured images are identified. Movement of edgesthat are counter to the sensed motion is ignored. Edges with leastamount of movement within the captured scene are determined to belocated at a higher distance (ex: infinity) from the display device.Edges with greater motion are determined to be closer to the displaydevice.

According to various example embodiments, occlusion objects within thescene are also detected. Occlusion objects correspond to objects locatedbetween a light-generating object and the display device and acts toblock some of the reflection exhibited on the display device.

Referring now to FIG. 5, therein illustrated is a flowchart of theoperational steps of an example method 124 for determining thereflection effect on the display device caused by thereflection-inducing zones identified at step 116.

At step 408, the position of a viewer viewing the display device isdetermined. Object recognition of the images captured of the scenefacing the display device can be performed to recognize the viewer. Inone example, the eyes of the viewer are located. The position of theviewer can be tracked over time.

At step 416, the reflection-inducing zones identified at 116 anddistance of each light-generating objects represented by thereflection-inducing zones are received.

At step 424, reflection zones are determined based on the position ofthe viewer and information pertaining to the reflection-inducing zonesand distances of the corresponding light-generating objects from displaydevice. The reflection zones that are determined represent thereflection exhibited on the display device as perceived by the usercaused by light-generating objects in the scene. It will be appreciatedthat the reflection zones may cover only a portion of the area of thedisplay device. For example, some areas of the display device do notcorrespond to a reflection-inducing zone and therefore are determined tonot exhibit reflection.

Referring now to FIG. 6, therein illustrated is a schematic diagramshowing the relative positions of a viewer, the display device and animage capture device. It will be understood that reflections on thedisplay device perceived by the viewer correspond to the viewer's viewof light-generating objects in the scene as seen from the virtual viewposition, which corresponds to the position of the viewer's eyesmirrored over the display device. However, the image capture device thatcaptures images of the scene is offset from the virtual view position.Accordingly, an extrapolation is applied to determine howlight-generating objects represented as reflection-inducing zones in theimages captured would be seen from the virtual view position.

According to one example embodiment, the reflection zones may berepresented as a veiling glare in the form of a 2-D array, whereinvalues in the array define the luminance value of the reflection zonesas perceived by the viewer on the display device.

Referring now to FIG. 7, therein illustrated is a flowchart of theoperational steps of an example method 132 for adjusting the targetimage to be displayed based on the reflection effect.

At step 508, the reflection zones determined at step 424 are received.The veiling glare defining luminance values of the reflection zone isalso received.

At step 516, the luminance values of the reflection zone are comparedwith luminance values of the target image to determine the adjustment tobe applied to the target image. For example, the luminance value of thereflection zone at a given area on the display device is compared withluminance value at a corresponding area (when displayed on the displaydevice) of the target image. The comparison of luminance values may becarried out on a pixel-by-pixel basis. Upsampling of the veiling glaremay applied.

According to one example embodiment, for one or more subareas of thetarget image where luminance values of the pixels of the target imageare greater than luminance values in corresponding subareas of thereflection zone, the luminance values of the pixels within the one ormore subareas areas are decreased.

For one or more subareas of the target image where luminance values ofthe pixels of the target image in the subareas are less than luminancevalues in corresponding subareas of the reflection zone, the luminancevalues of the pixels within the one or more subareas are increased.

It will be appreciated that this has the effect of reducing luminancevalues in the target image in areas where the target image already hashigh luminance values and boosting the luminance values in the targetimage in areas where the target images does not have luminance values.

Areas of the target image that correspond to areas of the display devicethat does not exhibit reflection (areas with no reflection zones asdetermined from steps 116 and 424) are not adjusted.

At step 524, the target image is adjusted according to the adjustmentcalculated at step 516.

At step 532, the adjusted target image is displayed on the displaydevice.

Example Implementation 1

The basic concept is to take continuous video, ambient light and motionsensor data from a mobile device, and use it to deduce in real-time thecurrent reflections seen by the user on his or her screen. The headposition is simultaneously tracked in order to reproject the brightregions as seen by the front-facing camera.

The main challenge with veil subtraction lies in the accurate predictionof the reflection image as seen from the viewer's perspective. If thereflection image is in the wrong place, the results may be deteriorateddue to subtracting the wrong veil, as shown in FIGS. 8a and 8b . Veilestimation requires (a) knowing where the viewer's eyes are located and(b) knowing the brightness and position of significant reflectionsources are, both relative to the display.

In one example implementation, it is assumed that the device being usedis equipped with a front-facing camera, an ambient light sensor, andmotion sensors to provide when and how the display is moving. Thesefeatures are typical on mobile devices currently, and are likely to beavailable with improved specifications in the future, but currentcapabilities are sufficient. An aim is to keep the computationalcomplexity low as well so as not to tax the CPU, GPU, or battery tooheavily.

The veil computation proceeds as follows:

-   -   Identify (track) head & eye position relative to screen. Use        intrinsic parameters of front-facing camera and typical human        head size to judge distance.    -   Find edges bordering bright zones in captured image. (See        example capture and bright area perimeter bitmap in FIGS. 9a and        9b .) Adjust threshold dynamically based on camera exposure such        that regions do not change too greatly in size.    -   When lateral movement is detected by motion sensors (not        rotation), identify edges in the bright area masks that are        roughly perpendicular to the direction of motion.    -   Edges that move strongly in the wrong direction are ignored as        they must be moving in the scene. Edges with the least motion        are assumed to be at infinity (vanishing point or horizon).        Edges with greater motion are assumed to be nearer, and their        distance is computed based on parallax against the horizon line        and intrinsic camera parameters.    -   With a build-up of small lateral motions in different        directions, gather information and confidence about the distance        to these bright area perimeters. Interpolate distances around        the perimeter and discard extreme outliers. Result is a set of        3-D contours enclosing highlight regions.    -   Use ambient sensor to estimate the average brightness of the        white (clipped) areas.    -   Reproject 3-D highlight contours based on estimated head/eye        position.    -   Multiply by screen BRDF (bidirectional reflection distribution        function) and blur based on assumed pupil size to get reflection        (veil) image estimate.    -   Apply the highlight mitigation technique(s) as described        elsewhere herein, choosing method & strength based on confidence        in the reflection estimate.

There are some important advantages to this method.

It can use a standard front-facing camera with exposure set too high toaccurately capture the brightness levels, because their intensity fromthe absolute levels recorded can be estimated from the adjacent ambientsensor.

Rather than solving the full 3-D reconstruction problem, bright areaperimeters are used to build up information from the device motionsensors and the unsteady hands of the viewer. Tracking edgesperpendicular to motion further reduces errors and avoids the expense offeature matching used in most depth-from-multiview methods.

Reprojection is also simplified, as a small number of discrete positionsare moved along a set of 3-D contours and in-filling to estimate newhighlights. This is designed as a lightweight process.

One effectively “trains” the user to move the display around a bit toimprove the image.

If future front-facing cameras have high dynamic-range capture capacity,the veil image will be further improved.

Highlight Thresholding

One task is to determine which parts of the captured front-camera viewconstitute possible highlights that will be reflected in the user'sscreen. A threshold is set empirically based on the captured image pixelvalue at the bridge of the user's nose. Since it is expected this willbe proportional to the viewer's eye adaptation and already in camerabrightness units, whatever the exposure happens to be, this serves as aconvenient way to set the threshold. A square area equal to theinter-ocular spacing is averaged, which covers most of the nose, some ofthe forehead, and roughly half of each eye. The highlight threshold isset to a multiple (ex: 100 times) of this average, or a factor (ex:0.92) of the maximum image value in a [0,1] range, whichever is smaller.Down-sampled capture image pixels that are above threshold using a 2-Dbitmap are marked. This bitmap is subsequently eroded and dilated toremove isolated highlights that would be too small to remedy.

Highlight Reprojection

For the reprojection step, the front camera geometry and determined headposition is used to extrapolate the view behind the screen thatcorresponds to the reflected image from the user's perspective. At thisstage, distance estimates for all the scene pixels is needed, which isobtained from a call-back function. In the demo for a car, this functionwill use the mock-up car's window geometry.

The virtual view that would not require any reprojection corresponds toan impossible position behind the display. Reprojecting highlightscaptured by the front camera depends on the distances to objects in thescene. The distance to the viewer's head outline can be estimated fromeye-tracking data, and other distances based on a fixed automobile demogeometry.

The diagram shown also simplifies the problem by showing only one eye.Since it is assumed that the viewer has two eyes, the reprojection isperformed twice and overlay the results. A 50% factor may be used foreach highlight after normalization based on the ambient sensor value.

To reproject pixels corresponding to different scene distances, pixelareas are opened up, using a quad-filling method. It is understood thatthere are portions of the reprojected view obstructed by the viewer'shead, where the highlights will be uncertain. These will be filled withhighlight or non-highlight regions, depending on the boundaries. Thefinal highlight image is then converted to absolute luminance asreflected in the display using a measured screen reflectance value.

Highlight Compensation

A significant problem with veil subtraction is that only so much can besubtracted from an image before “black” is reached, since it is notpossible to produce negative light. This is visible in the pinkishremnants of the original highlights in the sky, for example.Fortunately, the eye is fairly forgiving of haze in real life, which iswhy raising the black level is a reasonable thing to do. Raising theblack level in a perception-friendly manner puts more haze towards thetop of the image, yielding the headroom needed for greater reflectionsubtraction. The amount of haze that is added can be adjusted based onthe viewing conditions and user preference for optimal results.

Once the highlight luminance as seen by the viewer is estimated, thetarget display image to minimize degradation needs to be compensated.The overall strategy is to subtract the veil due to highlights wheretarget image values permit, and raise other values as necessary tosubtract the veil in target regions that would otherwise be too dark forsubtraction. This fits the goal of maintaining local contrast despitehighlights, at the expense of manipulating brightness in some highlightregions. The overall effect is an interesting one, which sits visuallybetween emissive and reflective displays. In brighter regions of thetarget image, where veil subtraction just works, the veil largelydisappears and colors are restored. In darker regions, the imagehighlight is like dappled light on a reflection print, bringing up localbrightness while preserving contrast and color appearance. This ends upbeing much less objectionable than other manipulations tested.

The highlight mitigation method can be described by the followingformula:

$L^{\prime} = \frac{{mL} - V}{k}$

where:

-   -   L′=adjusted value for pixel    -   L=original value for pixel    -   V=veiling glare

${m = {{1\mspace{14mu} {when}\mspace{14mu} L_{low}} \geq V_{avg}}},{{\min \left( {\frac{V_{avg}}{L_{low}},\frac{V_{avg} + k}{L_{high}}} \right)}\mspace{14mu} {otherwise}}$

-   -   V_(avg)=local average Y of veiling glare    -   L_(low)=minimum local Y value    -   L_(high)=maximum local Y value    -   k=headroom value, ≥1.0

Target image values are converted to single-channel floating-point in a[0,1] range. The L_(low) and L_(high) values are calculated within smallregions of the target image used to set the local multiplier m. Thisdown-sampled multiplier image as well as the veil image are smoothed(blurred) by a certain amount to avoid the appearance of artificialboundaries. The headroom constant k may be used to provide additionalrange on displays that can boost small regions to bright values, buthave difficulty maintaining high total output, such as OLED devices.Settings above 1.0 will slightly dim the display everywhere that addedbrightness is not needed to compensate for highlights.

Example Implementation 2 Creating a Demo for Automotive DisplayApplications

The automotive application is a constrained subproblem that avoids theneed to estimate the distances to highlight boundaries in the scene,since the rough geometry of the vehicle is known. The viewer's headposition still needs to be tracked based on front-camera data. Forexample, Android comes with built-in calls that perform this task.

The viewer's eye position together with the camera field of view andposition with respect to the display area are used to reproject brightregion locations to where they are expected to appear in the reflectedscreen. This step is performed on a pixel-by-pixel basis inover-threshold regions of the captured image, but at a reducedresolution to maintain responsiveness. In this situation, shifts in theviewer's head position and changes in the scene behind the viewer areresponded to. Some delay (ex: on the order of fraction of seconds) isacceptable.

The example implementation seeks to simulate an in-dash display roughlypositioned in front of the driver.

Port/Implement Head-Tracking Method to Android

Access front-facing camera and track head/eye position in real-time,predicting movement as needed to reduce latency to zero. This maybenefit from access to the motion sensors for mobile applications.

Develop/Test Method for Determining Highlight Brightness

Using front-facing camera in conjunction with ambient sensor, estimateillumination environment and compare to an HDR capture from the same(camera's) perspective. Evaluate and eliminate sources of error to theextent possible.

Determine Method for Entering Vehicle Geometry

Create simple user interface for entering rough distances and number ofwindows needed for demo performance.

Select Video Content for Demo

Select a simulated instrument panel, heads-up or other display contentappropriate to driver's-side automotive applications.

Design and Test Reprojection Technique

Using entered vehicle window geometry and tablet position, test accuracyof head-tracking combined with reflection prediction by tracing expected(reflected) window outlines in tablet display.

Choose Appropriate Highlight Mitigation Strategy

Test and tune highlight mitigation technique in different lightingsituations and choose most effective method(s).

Add Ability to Detect Passengers

Modify app behavior based on likely passenger occlusion in scenario ofdriver with adjacent front-seat passenger.

Example Implementation 3 Modifying Demo for Mobile ApplicationsSingle-Viewer Head-Tracking

With input from orientation sensor and front-facing camera, trackviewer's head based on likely/possible viewing configurations. Usehead-size and/or eye spacing to estimate distance and associatedsilhouette boundary.

Digesting Motion-Sensor Input

Access motion-sensor and develop robust method for detecting “parallaxshifts” in plane of display as needed for boundary distancedetermination. Learn to discriminate between in-plane shifts and themore common rotations that occur as a user holds the device naturallywhile interacting and viewing content. Use recorded front-facing videoto evaluate performance of motion-tracking analysis.

Estimating Highlight Region Boundaries

Using thresholding technique, identify bright region boundaries and useperpendicular in-plane motion to estimate distances and connect contoursas needed where distances are unreliable. Check against measurements.

Live Tracking

Use 3-D highlight region boundary and luminance estimates together withhead-tracking to reproject contour lines in live display andtest/improve accuracy. Derive confidence values from motion and camerainput.

Demo Integration

Couple reflection estimates including intensity with mitigationdeveloped for automotive demo to complete method. Test with a variety ofimage and video content.

Frame-Rate Tuning

Using motion data and confidence values, reduce frame-rate whenreflections are not a problem or confidence is high and no head, sceneor device motion is detected.

The current demo implementation consists of two C++ classes that run ona Samsung tablet device, and a Java interface with GPU code to interfacewith Android and the graphics hardware. Here is the list of input valuesto obtain prior to calling C++ classes:

(a) Static parameters such as display resolution, size, screenreflectance, front camera geometry and intrinsics (FOV, etc.)

(b) Dynamically captured image from the front camera

(c) Ambient sensor reading (assume similar view to camera)

(d) Eye position (bridge of nose) and separation (interocular distance)measured in front camera pixels

(e) Estimated distances to pixels in front camera image (from vehiclegeometry and head position)

(f) The target image or video frame for display

The first C++ class is GetHighlights, and its purpose is to determinethe position and relative intensity of reflected highlights visible tothe viewer. All of the above information is needed except for the targetdisplay image (f), which is applied in another class, HideHighlights.The GetHighlights class performs the following operations on the input:

(1) Downsample the captured front image into a low-resolution (roughly400×400 pixel) version

(2) Determine a highlight intensity threshold based on the observed nosebrightness as described in Method section

(3) Find regions of the low-resolution image above this threshold, thenerode and dilate by 2 down-sampled pixels to remove small features

(4) Remap highlights by projecting them into each eye's view as seen inthe reflected screen

(5) Correct reflection magnitude based on ambient sensor reading, screenreflectance, and average brightness of highlights

It is not assumed the front camera image is HDR or even calibrated inany useful way. In fact, many of the highlight regions may be clipped tothe maximum image value. The normalization in step (5) is specificallydesigned to compensate for this lack of calibration, substituting theabsolute value from the ambient sensor and inferring that most of themeasured light is represented somewhere in the image, even if it showsonly as white. By scaling the highlights by the ambient sensor reading,the recorded highlights are obtained back into roughly the correctrange. The only assumption is that the exposure is bright enough totrack the user's eye positions and dim enough that everything else isnot completely blown out. In cases where there is nothing significantlyabove the brightness of the viewer's face, no highlights will bereturned and the loop can be paused until the ambient sensor detects achange in the lighting.

The result produced by the GetHighlights class is a low-resolution imagematching the display's aspect ratio with the highlights the viewer isexpected to see reflected at the moment the front image was captured.This estimated highlight image then gets passed along with the targetdisplay image (f) to the HideHighlights class to perform the followingsteps:

(6) Compute a multiplier at each display pixel that brightens up theimage enough to subtract the offending highlights as described in Methodsection

(7) Subtract the highlight image from the target display image aftermultiplication

For step (6), how pixel values translate to absolute luminance on thedisplay needs to be known. This should be determined by the brightnesssetting available in Android, but there seems to be a complex, dynamicrelation between this setting and the actual pixel luminances on theOLED display.

The final operation (7) is the only one performed at full displayresolution, but can be sped up easily enough by employing the GPU.

Several alternative embodiments and examples have been described andillustrated herein. The embodiments of the invention described above areintended to be exemplary only. A person skilled in the art wouldappreciate the features of the individual embodiments, and the possiblecombinations and variations of the components. A person skilled in theart would further appreciate that any of the embodiments could beprovided in any combination with the other embodiments disclosed herein.It is understood that the invention may be embodied in other specificforms without departing from the central characteristics thereof. Thepresent examples and embodiments, therefore, are to be considered in allrespects as illustrative and not restrictive, and the invention is notto be limited to the details given herein. Accordingly, while specificembodiments have been illustrated and described, numerous modificationscome to mind without significantly departing from the scope of theinvention as defined in the appended claims.

1. A method for compensating for reflection on a display device, themethod comprising: capturing one or more images of a scene facing thedisplay device; identifying from the captured images one or morereflection-inducing zones located within the scene facing the displaydevice; determining specular reflection effect on the display devicecaused by the reflection-inducing zones; and adjusting a target image tobe displayed on the display device based on the determined specularreflection effect.
 2. The method of claim 1, wherein adjusting thetarget image comprises adjusting luminance values of a plurality ofimage pixels within one or more regions of the target image; and whereinthe regions of the target image are determined from the reflectioninducing-zones.
 3. (canceled)
 4. (canceled)
 5. The method of claim 1,wherein the one or more reflection-inducing zones are determined basedon luminance values of pixels within the one or more captured images. 6.The method of claim 5, wherein the one or more reflection-inducing zonescorrespond to areas within the one or more captured images having aluminance value greater than a predetermined threshold.
 7. The method ofclaim 6, wherein capturing the one or more images of the scene facingthe display device comprises capturing the face of a viewer locatedwithin the scene; and wherein the predetermined threshold is determinedbased on a luminance value of a zone within the one or more capturedimages corresponding to a portion of the face of the viewer.
 8. Themethod of claim 1, further comprising determining from the capturedimages a position of the viewer viewing the display device; and whereinthe reflection effect on the display device caused by thereflection-inducing zones is determined based on the position of theviewer and the distance of each of the reflection-inducing zones fromthe display device.
 9. (canceled)
 10. The method of claim 1, furthercomprising determining a distance of each of the reflection-inducingzones from the display device.
 11. The method of claim 10, wherein thedistance of one or more of the reflection-inducing zones from thedisplay device is determined based on predefined properties oflight-generating objects located within the scene; and wherein thepredefined properties of a given one of the light-generating objectslocated within the scene comprise one or more of location, predefinedluminance values, and size.
 12. (canceled)
 13. (canceled)
 14. The methodof claim 11, further comprising identifying occluding elements locatedwithin the scene.
 15. (canceled)
 16. (canceled)
 17. (canceled) 18.(canceled)
 19. The method of claim 1, wherein the reflection effectdefines location and luminance values of images pixels of one or morereflection zones exhibited on the display device due toreflection-inducing zones located within the scene facing the displaydevice; and wherein adjusting the target image comprises adjustingluminance values within the target image in accordance to the determinedreflection effect.
 20. (canceled)
 21. (canceled)
 22. The method of claim19, wherein adjusting luminance values comprises for an area of thetarget image corresponding to a given one of the one or more reflectionzones: for one or more subareas of the area where luminance values ofthe pixels of the target image are greater than luminance values incorresponding subareas of the reflection zone, reducing the luminancevalues of the pixels within the one or more subareas; and for one ormore subareas of the area where luminance values of the pixels of thetarget image in the subareas are less than luminance values incorresponding subareas of the reflection zone, increasing the luminancevalues of the pixels within the one or more subareas.
 23. The method ofclaim 1, wherein the one or more images of the scene facing the displaydevice are captured by a front facing camera offset by a known distanceand orientation from the display device.
 24. (canceled)
 25. (canceled)26. The method of claim 1, wherein adjusting the target image is furtherbased on a predefined reflectance of the display device.
 27. Acomputer-implemented system comprising: at least one data storagedevice; and at least one processor operably coupled to the at least onestorage device, the at least one processor being configured for:receiving one or more captured images of a scene facing the displaydevice; identifying from the captured images one or morereflection-inducing zones located within the scene facing the displaydevice; determining specular reflection effect on the display devicecaused by the reflection-inducing zones; and adjusting a target image tobe displayed on the display device based on the determined specularreflection effect.
 28. The system of claim 27, wherein adjusting thetarget image comprises adjusting luminance values of a plurality ofimage pixels within one or more regions of the target image; and whereinthe regions of the target image are determined from the reflectioninducing-zones.
 29. (canceled)
 30. (canceled)
 31. The system of claim27, wherein the one or more reflection-inducing zones are determinedbased on luminance values of pixels within the one or more capturedimages.
 32. The system of claim 31, wherein the one or morereflection-inducing zones correspond to areas within the one or morecaptured images having a luminance value greater than a predeterminedthreshold.
 33. The system of claim 32, wherein the one or more capturedimages of the scene facing the display device comprise one or morecaptured images of the face of a viewer located within the scene; andwherein the predetermined threshold is determined based on a luminancevalue of a zone within the one or more captured images corresponding toa portion of the face of the viewer.
 34. The system of claim 27, whereinthe processor is further configured for determining from the capturedimages a position of the viewer viewing the display device; and whereinthe reflection effect on the display device caused by thereflection-inducing zones is determined based on the position of theviewer and the distance of each of the reflection-inducing zones fromthe display device.
 35. (canceled)
 36. The system of claim 27, whereinthe processor is further configured for determining a distance of eachof the reflection-inducing zones from the display device.
 37. The systemof claim 36, wherein the distance of one or more of thereflection-inducing zones from the display device is determined based onpredefined properties of light-generating objects located within thescene; and wherein the predefined properties of a given one of thelight-generating objects located within the scene comprise one or moreof location, predefined luminance values, and size.
 38. (canceled) 39.(canceled)
 40. The system of claim 37, wherein the processor is furtherconfigured for identifying occluding elements located within the scene.41. (canceled)
 42. (canceled)
 43. (canceled)
 44. (canceled)
 45. Thesystem of claim 27, wherein the reflection effect defines location andluminance values of image pixels of one or more reflection zonesexhibited on the display device due to reflection-inducing zones locatedwithin the scene facing the display device; and wherein adjusting thetarget image comprises adjusting luminance values within the targetimage in accordance to the determined reflection effect.
 46. (canceled)47. (canceled)
 48. The system of claim 45, wherein adjusting luminancevalues comprises for an area of the target image corresponding to agiven one of the one or more reflection zones: for one or more subareasof the area where luminance values of the pixels of the target image aregreater than luminance values in corresponding subareas of thereflection zone, reducing the luminance values of the pixels within theone or more subareas; and for one or more subareas of the area whereluminance values of the pixels of the target image in the subareas areless than luminance values in corresponding subareas of the reflectionzone, increasing the luminance values of the pixels within the one ormore subareas.
 49. The system of claim 27, wherein the one or moreimages of the scene facing the display device are captured by a frontfacing camera offset by a known distance and orientation from thedisplay device.
 50. (canceled)
 51. (canceled)
 52. The system of claim27, wherein adjusting the target image is further based on a predefinedreflectance of the display device.
 53. A non-transitory computerreadable storage medium comprising computer executable instructions,which when executed by at least one processor causes the at least oneprocessor to perform: capturing one or more images of a scene facing thedisplay device; identifying from the captured images one or morereflection-inducing zones located within the scene facing the displaydevice; determining specular reflection effect on the display devicecaused by the reflection-inducing zones; and adjusting a target image tobe displayed on the display device based on the determined specularreflection effect.